Skip to content

FEAT: Prompt generation jobs in the backend (-5) - #4134

Merged
Richard Lundeen (richlundeen) merged 57 commits into
microsoft:mainfrom
richlundeen:richlundeen-assert-generation-backend
Nov 19, 2026
Merged

Richard Lundeen (richlundeen) merged 57 commits into
microsoft:mainfrom
richlundeen:richlundeen-assert-generation-backend

Conversation

@richlundeen

@richlundeen Richard Lundeen (richlundeen) commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Part -5 of a 10-PR ASSERT stack. Design and how to use it: ASSERT in PyRIT.

PR
-1 #4132 Judge dimensions, presets and shared techniques merged
-2 #4130 Trace evidence (OpenTelemetry) merged
-3 #4131 Headline metrics merged
-4 #4133 Staged generation (SDK) merged
-5 #4134 Generation jobs (backend) this PR
-6 #4135 Generate from the GUI
-7 #4144 In-process agents (docs)
-8 #4180 Scenario-aware dataset compatibility
-9 #4226 ASSERT results grouped by generated category
-10 #4228 Background dataset loading (all scanners)

Parts -1 to -4 are merged, so this PR's Files changed tab shows only its own changes. Each later PR targets main and includes the unmerged layers before it.

Description

The backend runs prompt generators as background jobs, so the GUI can estimate, run, review, resume, and save generated datasets. It isn't tied to ASSERT: the target-objective generator uses the same jobs.

  • PromptGenerationService owns the job lifecycle: admission (one running generation per backend), checkpoint persistence, review pauses, cancel, retry from the last checkpoint, batch creation for suites, and saving a new named dataset. Shutdown lets a dataset save finish. Continue and retry need only the targets of the stages that remain. pause_for_review is rejected for generators without a review step.
  • Adapters translate the ASSERT and target-objective generators into the service's generic records. The generators own model calls, stage plans, and checkpoints, and the dataset providers package the results.
  • Records are PromptGenerationEntries rows in memory. The new Alembic revision c4a401cd8201 creates the table, and ccf97e771f04 adds each generation's batch position. SQLite memory applies both at startup; Azure SQL deployments apply them with pyrit_migrate.
  • Dataset names follow one create-only rule that uploads share: a name is taken while stored seeds use it, an upload is staging it, or an unfinished generation holds it. Saving creates the dataset and completes the record in one transaction; until then the generation stays running.
  • A target-objective dataset stores its request through TargetObjectiveGenerationRequest (metadata key pyrit_target_objective_request), so Edit & regenerate can read it back. TargetObjectiveProvider.package_result takes that request as its one source, including the harm categories that label the seeds, in place of its harm_categories argument; the signature came from -4 and is in no release. TargetObjectiveGenerator validates its inputs through the same model.
  • AssertGenerator.validate_checkpoint checks, without model calls, that the targets can run a checkpoint's remaining stages. advance_async runs the same check, and the backend uses it before it queues a continue or retry.
  • Routes:
    • /api/prompt-generators: generator descriptors and the bundled ASSERT behaviors and suites.
    • /api/prompt-generations: preflight with estimate, create, review, continue/retry/cancel, delete a record, and the request recorded in an ASSERT or objective dataset.
    • /api/prompt-generations/batch: queues generations that share settings, such as an ASSERT suite, as ordinary records with a shared group_id. Each member records its position in the request, which orders members that share a creation time.
  • DatasetService and the generation service share one parse_dataset_selection_key, including the dataset delete route, without changing which selection keys are accepted.
  • The backend README describes the routes and the caller-visible rules.

Storage and concurrency. Each record change is a compare-and-set on the statuses it expects, and it writes only the fields it changes, so a status change can't revert a saved review or another backend's checkpoint. A run claims its generation with a new execution ID, and its checkpoint, review, save, and failure writes require that ID. A cancel or retry from another backend therefore stops the previous owner at its next write. Backends that share one database see the same records and the same name rule. After a restart, SQLite memory marks generations that were active as interrupted; shared memory, such as Azure SQL, leaves them active until they are cancelled, because another backend may own them.

GUI and scope

Own GUI changes: none. This layer changes no frontend files; its routes back the GUI added in #4135.

Shared / non-ASSERT impact (backend only):

  • The same job lifecycle, routes and records serve ordinary target-objective generation, not only ASSERT.
  • Generations save ordinary named datasets in memory. Deleting a generation record doesn't delete its saved dataset.
  • Dataset uploads also return 409 for a name that an unfinished generation holds.
  • Dataset reads and deletes now go through the shared selection-key parser.

Unchanged / deferred:

  • Generation records name each role's target by registry name. A continue or retry resolves the names of the roles that still have work, and the checkpoint rejects a target whose identifier differs from the one that did the earlier work.
  • Edit & regenerate creates a new generation and a new dataset; there's no in-place dataset edit, versioning, archiving or retention.

Adds ASSERT judge dimension/preset models, ships ASSERT's 11 judge presets, judges custom boolean and ordinal dimensions (with not-applicable) through build_assert_policy_scorer, aggregates repeated judge samples as ASSERT does, and exposes judge_presets, disabled_judge_dimensions, judge_samples, and objective_dimension on foundry.assert_policy. Default identities are unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ge presets to built-in names

- objective_dimension and other non-default judge settings are recorded in scenario identity params, so resume detects changes and saved GUI/API launches restore them. Default identities are unchanged.
- Mixed boolean/ordinal judge sets are ordered by policy for a stable identity.
- judge_presets accepts only shipped preset names; preset files move to the Python-only judge_preset_files (list[Path]) parameter, which the API guard rejects. Preset load errors no longer echo file contents.
- Adds a Scenario._get_identity_params hook.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 10, 2026 06:30
@richlundeen
Richard Lundeen (richlundeen) marked this pull request as draft November 10, 2026 06:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical storage and cross-backend coordination races can overwrite terminal state, hide jobs, and release dataset-name reservations while original workers remain active, allowing duplicate generation.

5 open issues
What changed in this PR

This PR adds backend-managed prompt generation jobs so clients can draft, estimate, review, resume, and save generated datasets using PyRIT’s existing generators and dataset providers.

Changes:

  • Integrates generation jobs with backend routing, runtime lifecycle, estimates, and dataset packaging.
  • Bundles ASSERT behavior and suite presets with attribution.
  • Adds supporting ASSERT scoring, trace evidence, analytics, frontend integration, and documentation.
File Description
THIRD_PARTY_NOTICES.txt Updates ASSERT attribution.
tests/​unit/​executor/​promptgen/​test_target_objective_generator.py Tests generation estimates.
tests/​unit/​datasets/​test_target_objective_provider.py Tests result packaging.
tests/​unit/​common/​test_lazy_package_imports.py Updates analytics export count.
tests/​unit/​assert_mocks.py Adds checkpoint fixtures.
tests/​unit/​analytics/​test_rate_statistics.py Tests binary and macro rates.
pyrit/​score/​true_false/​true_false_fallback_scorer.py Adds configurable fallback triggers.
pyrit/​score/​scorer.py Identifies missing-evidence results.
pyrit/​score/​observation/​otel_span_exporter.py Exports richer trace evidence.
pyrit/​score/​observation/​message_trace_resolver.py Resolves conversation trace links.
pyrit/​score/​observation/​execution.py Shares evidence acquisition across scorers.
pyrit/​score/​observation/​__init__.py Exports trace observation APIs.
pyrit/​score/​float_scale/​float_scale_fallback_scorer.py Adds configurable fallback triggers.
pyrit/​score/​__init__.py Exports ASSERT and tracing APIs.
pyrit/​scenario/​core/​scenario.py Extracts scenario identity parameters.
pyrit/​prompt_target/​common/​target_trace_config.py Adds conversation identifiers to spans.
pyrit/​prompt_target/​common/​prompt_target.py Exposes target tracing configuration.
pyrit/​models/​score/​__init__.py Exports execution-trace models.
pyrit/​models/​prompt_generation_estimate.py Defines generation estimate contracts.
pyrit/​models/​analytics.py Adds rate result models.
pyrit/​executor/​promptgen/​target_objective_generator.py Estimates objective-generation work.
pyrit/​executor/​promptgen/​__init__.py Exports objective-generation results.
pyrit/​datasets/​seed_datasets/​generated/​target_objective_provider.py Packages existing generation results.
pyrit/​datasets/​score/​true_false_question/​assert_execution_evidence_instructions.yaml Guides execution-evidence judgments.
pyrit/​datasets/​score/​assert/​judge_presets/​tool-use.yaml Adds tool-use judging criteria.
pyrit/​datasets/​score/​assert/​judge_presets/​safety-extended.yaml Adds extended safety criteria.
pyrit/​datasets/​score/​assert/​judge_presets/​safety-core.yaml References core safety dimensions.
pyrit/​datasets/​score/​assert/​judge_presets/​robustness.yaml Adds robustness judging criteria.
pyrit/​datasets/​score/​assert/​judge_presets/​policy-adherence.yaml Adds policy-adherence criteria.
pyrit/​datasets/​score/​assert/​judge_presets/​operational.yaml Adds operational judging criteria.
pyrit/​datasets/​score/​assert/​judge_presets/​multi-turn.yaml Adds multi-turn judging criteria.
pyrit/​datasets/​score/​assert/​judge_presets/​instruction-following.yaml Adds instruction-following criteria.
pyrit/​datasets/​score/​assert/​judge_presets/​grounding.yaml Adds grounding judging criteria.
pyrit/​datasets/​score/​assert/​judge_presets/​communication.yaml Adds communication judging criteria.
pyrit/​datasets/​score/​assert/​judge_presets/​alignment.yaml Adds alignment judging criteria.
pyrit/​datasets/​score/​assert/​dimension_system_prompt.yaml Adds custom-dimension judge template.
pyrit/​datasets/​assert_suites/​travel_planner.yaml Bundles travel-planner suite.
pyrit/​datasets/​assert_suites/​travel_planner_benchmark.yaml Bundles travel-quality benchmark suite.
pyrit/​datasets/​assert_suites/​telecom_customer_service.yaml Bundles telecom support suite.
pyrit/​datasets/​assert_library.py Loads and validates bundled presets.
pyrit/​datasets/​assert_behaviors/​violent_content.yaml Bundles violence behavior preset.
pyrit/​datasets/​assert_behaviors/​unsupported_conclusion_failures.yaml Bundles unsupported-conclusion preset.
pyrit/​datasets/​assert_behaviors/​unit_conversion_failures.yaml Bundles unit-conversion preset.
pyrit/​datasets/​assert_behaviors/​underused_context_failures.yaml Bundles context-underuse preset.
pyrit/​datasets/​assert_behaviors/​unauthorized_action_failures.yaml Bundles unauthorized-action preset.
pyrit/​datasets/​assert_behaviors/​tool_parameter_formatting_failures.yaml Bundles tool-argument preset.
pyrit/​datasets/​assert_behaviors/​tool_output_misinterpretation_failures.yaml Bundles tool-output interpretation preset.
pyrit/​datasets/​assert_behaviors/​tool_orchestration_errors.yaml Bundles tool-orchestration preset.
pyrit/​datasets/​assert_behaviors/​tool_call_turn_protocol_failures.yaml Bundles tool-turn protocol preset.
pyrit/​datasets/​assert_behaviors/​tool_call_error_recovery_failures.yaml Bundles tool-error recovery preset.
pyrit/​datasets/​assert_behaviors/​tabs_vs_spaces.yaml Bundles indentation-bias preset.
pyrit/​datasets/​assert_behaviors/​sycophancy.yaml Bundles sycophancy preset.
pyrit/​datasets/​assert_behaviors/​suicide_self_harm.yaml Bundles self-harm safety preset.
pyrit/​datasets/​assert_behaviors/​success_criteria_ambiguity_failures.yaml Bundles success-criteria preset.
pyrit/​datasets/​assert_behaviors/​stereotyping.yaml Bundles stereotyping preset.
pyrit/​datasets/​assert_behaviors/​stale_state_failures.yaml Bundles stale-state preset.
pyrit/​datasets/​assert_behaviors/​shimano_vs_sram.yaml Bundles cycling-brand bias preset.
pyrit/​datasets/​assert_behaviors/​sexual_content.yaml Bundles sexual-content safety preset.
pyrit/​datasets/​assert_behaviors/​response_completeness_failures.yaml Bundles response-completeness preset.
pyrit/​datasets/​assert_behaviors/​repeated_action_loop_failures.yaml Bundles repeated-action preset.
pyrit/​datasets/​assert_behaviors/​prompt_injection.yaml Bundles prompt-injection preset.
pyrit/​datasets/​assert_behaviors/​procedure_adherence_failures.yaml Bundles procedure-adherence preset.
pyrit/​datasets/​assert_behaviors/​premature_termination_failures.yaml Bundles premature-termination preset.
pyrit/​datasets/​assert_behaviors/​poor_retrieval_failures.yaml Bundles retrieval-quality preset.
pyrit/​datasets/​assert_behaviors/​output_internal_consistency_failures.yaml Bundles output-consistency preset.
pyrit/​datasets/​assert_behaviors/​out_of_scope_request_failures.yaml Bundles scope-compliance preset.
pyrit/​datasets/​assert_behaviors/​observation_neglect_failures.yaml Bundles observation-neglect preset.
pyrit/​datasets/​assert_behaviors/​malicious_cyber_activity.yaml Bundles cyber-safety preset.
pyrit/​datasets/​assert_behaviors/​ios_vs_android.yaml Bundles platform-bias preset.
pyrit/​datasets/​assert_behaviors/​inter_agent_handoff_failures.yaml Bundles agent-handoff preset.
pyrit/​datasets/​assert_behaviors/​intent_misinterpretation_failures.yaml Bundles intent-interpretation preset.
pyrit/​datasets/​assert_behaviors/​insufficient_verification_failures.yaml Bundles verification-failure preset.
pyrit/​datasets/​assert_behaviors/​ineffective_team_communication_failures.yaml Bundles agent-communication preset.
pyrit/​datasets/​assert_behaviors/​incorrect_tool_selection_failures.yaml Bundles tool-selection preset.
pyrit/​datasets/​assert_behaviors/​incomplete_answer_synthesis_failures.yaml Bundles answer-synthesis preset.
pyrit/​datasets/​assert_behaviors/​identity_verification_failures.yaml Bundles identity-verification preset.
pyrit/​datasets/​assert_behaviors/​hate_speech_harassment.yaml Bundles harassment-safety preset.
pyrit/​datasets/​assert_behaviors/​harmful_medical_advice.yaml Bundles medical-safety preset.
pyrit/​datasets/​assert_behaviors/​grounding_attribution_errors.yaml Bundles grounding-error preset.
pyrit/​datasets/​assert_behaviors/​goal_drift_failures.yaml Bundles goal-drift preset.
pyrit/​datasets/​assert_behaviors/​flawed_action_plan_failures.yaml Bundles action-planning preset.
pyrit/​datasets/​assert_behaviors/​explicit_constraint_violation_failures.yaml Bundles constraint-compliance preset.
pyrit/​datasets/​assert_behaviors/​escalation_judgment_failures.yaml Bundles escalation-decision preset.
pyrit/​datasets/​assert_behaviors/​doxxing.yaml Bundles privacy-safety preset.
pyrit/​datasets/​assert_behaviors/​crystal_mountain_vs_snoqualmie.yaml Bundles destination-bias preset.
pyrit/​datasets/​assert_behaviors/​conversation_coherence_breakdown.yaml Bundles conversation-coherence preset.
pyrit/​datasets/​assert_behaviors/​constraint_propagation_failures.yaml Bundles constraint-propagation preset.
pyrit/​datasets/​assert_behaviors/​conflicting_instruction_resolution_failures.yaml Bundles instruction-conflict preset.
pyrit/​datasets/​assert_behaviors/​actionability_failures.yaml Bundles actionability preset.
pyrit/​datasets/​__init__.py Exports preset lookup APIs.
pyrit/​common/​path.py Defines bundled preset paths.
pyrit/​backend/​services/​service_lifecycle.py Resets generation service during cleanup.
pyrit/​backend/​services/​runtime_lifecycle.py Includes generation jobs in lifecycle checks.
pyrit/​backend/​routes/​scenarios.py Exposes ASSERT metrics endpoint.
pyrit/​backend/​README.md Documents jobs and metrics APIs.
pyrit/​backend/​main.py Registers generation routes.
pyrit/​analytics/​__init__.py Exports ASSERT analytics helpers.
frontend/​src/​types/​index.ts Defines ASSERT metrics contracts.
frontend/​src/​services/​api.ts Adds metrics API client.
frontend/​src/​hooks/​useAssertPolicyMetrics.ts Loads metrics with request cancellation.
frontend/​src/​hooks/​useAssertPolicyMetrics.test.tsx Tests metrics loading and retries.
frontend/​src/​components/​Scenarios/​ScenarioRunPage.tsx Displays metrics for completed ASSERT runs.
frontend/​src/​components/​Scenarios/​ScenarioRunPage.test.tsx Tests conditional metrics display.
frontend/​src/​components/​Scenarios/​assertPolicyRun.ts Identifies ASSERT policy runs.
frontend/​src/​components/​Scenarios/​AssertPolicyMetricsPanel.styles.ts Styles metrics presentation.
doc/​myst.yml Adds trace-scoring navigation.
doc/​code/​scoring/​1_true_false_scorers.py Documents dimensions and sampled judgments.
doc/​code/​scoring/​1_true_false_scorers.ipynb Updates paired scoring documentation.
doc/​code/​scoring/​0_scoring.py Links execution-trace scoring guidance.
doc/​code/​scoring/​0_scoring.ipynb Updates paired scoring overview.
doc/​code/​framework.md Defines job, trace, and analytics responsibilities.
doc/​code/​executor/​7_promptgen.py Demonstrates checkpoint review and continuation.
doc/​code/​executor/​7_promptgen.ipynb Adds paired generation example.
doc/​code/​analytics/​0_attack_results.md Documents rate helpers and ASSERT metrics.

Review effort: Balanced

…converters

- Custom judge dimensions are boolean bad-event flags judged by SelfAskTrueFalseScorer with an ASSERT dimension template; ASSERT's ordinal scale and allow_not_applicable options are rejected.
- Repeated samples use TrueFalseCompositeScorer over the repeated judge with an ASSERT vote (ties count as failures, undetermined samples excluded, category re-vote with override warning), replacing the bespoke sampled scorers.
- Public exports trimmed to AssertJudgeDimension, AssertJudgePreset, AssertSampleMetadata, and the preset list/load functions.
- foundry.assert_policy accepts technique_converters (none by default) for both techniques.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- AssertTechnique is removed. The technique class is built from the AttackTechniqueRegistry, with red_teaming as the default (ASSERT's scenario case).
- The baseline, now on by default, sends each group's stored system and user prompts (ASSERT's prompt case).
- max_turns applies to techniques whose attack takes a turn count; results are grouped by technique.
- AssertPolicy.VERSION is 2, so runs from version 1 do not resume.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- allow_not_applicable: a not-applicable answer is undetermined with assert_not_applicable metadata; repeated samples exclude it and keep the marker only when every sample is not applicable.
- ASSERT's ordinal scale is accepted with a PyRIT violation_at grade. The judge's grade position is thresholded by FloatScaleThresholdScorer, so every dimension still yields a true/false verdict, and the grade is kept in assert_grade metadata. Graded dimensions can be the objective.
- AssertDimensionMetadata is exported from pyrit.score. Default identities are unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

@romanlutz Roman Lutz (romanlutz) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the incremental diff vs the -4 branch (richlundeen-assert-generation-sdk). Solid job layer — the lifecycle edge cases (shielded cancel updates, shutdown waiting for saves, idempotent retry-after-save) are handled with care, and the tests exercise real generators. One small 500-level bug and a handful of polish items below. All 37 new tests pass for me locally.

Adds a scenario-local assert_package technique to foundry.assert_policy: PyRIT's RedTeamingAttack whose tester uses ASSERT's own tester system prompt (pinned 04b713fe) filled with each group's stored tester instructions. It is the default technique alongside the baseline; red_teaming stays selectable. Groups without tester instructions are skipped or rejected per incompatible_technique_policy.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Resolve judge presets in a worker thread instead of the synchronous validator. Drop a judge sample whose judgment stays invalid after retries (scoring fails only when every sample does), as ASSERT does. Require native multi-turn targets for turn-taking techniques in foundry.assert_policy, keep turn settings a registered technique sets itself, pass the adversarial chat only to techniques that use one, and warn once per even sample count.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@richlundeen
Richard Lundeen (richlundeen) force-pushed the richlundeen-assert-generation-backend branch from ea4dbc1 to 6f66fd2 Compare November 10, 2026 23:18

@romanlutz Roman Lutz (romanlutz) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review of ea4dbc1. The simplification pass (drafts, versioning, and the groups API removed; single-backend ownership made explicit) resolves my round-1 points 1, 4, 5, and 6 — point 1's 500 bug went away with the PATCH endpoint itself — and the new cancel-before-start handling fixes a latent bug I'd missed. Still open from round 1: pause_for_review accepted for non-reviewable generators, and the exact preset counts in the route test. Three small new items inline. All 35 tests pass locally on the new head.

Add AssertDimensionResult, AssertCategoryJudgment (with AssertNodeJudgment) and AssertSampleVotes in pyrit.models as the only readers and writers of the ASSERT score metadata keys, keeping the stored key format. Missing keys are unset and malformed present data raises ValueError. The scenario judge wrapper now records the assert_dimension marker on judged and skipped scores. The custom-dimension judge renders its request from a YAML SeedPrompt, and both ASSERT response handlers validate and convert the decoded response through a new JsonSchemaResponseHandler._convert_response hook instead of parsing twice. AssertDimensionMetadata and AssertSampleMetadata are removed.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ence

ASSERT judges can read what the application did, not only what it said: tool
calls with arguments and results, model calls, and agent steps captured with
OpenTelemetry, rendered after the turns they produced with citable step IDs.

- LocalTraceCapture: in-process provider and store that keeps target-request
  traces, completes them when their spans end, and evicts the oldest when full.
- OtelExecutionTraceSource: OpenInference and GenAI normalization into a durable
  execution_trace observation with bounded, redactable content.
- ConversationScorer trace_source and trace_requirement; AssertTraceEvidence for
  build_assert_policy_scorer; foundry.assert_policy trace_source and
  trace_requirement, kept in identity when non-default.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Withdraw completion when a span starts after it; export and completion share the capture lock; evict any trace when full with explicit loss markers and bounded bookkeeping; render steps in parent cycles; remove the unreachable unattributed block; require a trace source for trace_requirement; document strict snapshot matching and citation behavior.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…mplate

additional_instructions is an explicit dimension_system_prompt.yaml parameter rendered in the sandbox instead of appended after rendering; the prompt is unchanged when it is empty.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Report ASSERT's permissible/impermissible split, a rate per boolean judge dimension, and
judged-category violation rates per technique, with 90% Wilson intervals and ASSERT's
minimum support. Expose them through a backend route and the GUI run page.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Integrates -3 d0b7c2a (typed judgments, provenance and verdict corrections) under the published -4 head; no conflicts.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Integrates -4 1c0add0 (rebuilt -3 d0b7c2a) under the published -5 head; no conflicts.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@richlundeen
Richard Lundeen (richlundeen) marked this pull request as ready for review November 18, 2026 05:56
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main's check-keyword-only-params hook flags four generation-job
definitions with two positional parameters. The create routes take
the user keyword-only, the checkpoint callback becomes a Protocol with
keyword-only progress, and the store's file replacement takes its data
keyword-only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Store prompt generation records in a new PromptGenerationEntries memory
table instead of JSON files with a read-once cache, so backends that share
a database see the same generations and every write is coordinated by the
database.

- Move PromptGenerationRecord, PromptGenerationStatus, the status sets,
  PromptGenerationProgress, and TargetObjectiveGenerationRequest to
  pyrit.models. Record ids and group ids are UUIDs; REST views still
  serialize them as strings and never expose the new execution_id.
- Add native-async memory methods: add, list (created_at DESC, id),
  compare-and-set update fenced by an optional execution id, compare-and-
  delete, name availability, and a commit that creates the dataset
  create-only and completes the record in one transaction under the
  dataset lifecycle lock. Migration c4a401cd8201 adds the table.
- Apply one create-only dataset name rule, shared by uploads and
  generations: stored seeds, staging uploads, and unfinished generations
  hold a name. Upload reservation now rejects generation-held names.
- The service compare-and-sets every transition and gives each run a fresh
  execution id that fences its checkpoint, review, save, and failure
  writes, so a cancel or retry from another backend stops the old owner
  instead of being overwritten. Cancelling works without a local task.
- Reconcile interrupted generations only for SQLite memory, like scenario
  runs. Remove PromptGenerationStore, _dataset_came_from_async, and the
  adapters' dataset_matches_checkpoint, which the atomic commit replaces.
- Remove the PromptGenerationService bullet from framework.md and describe
  storage and concurrency in the backend README.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ames

An orphaned SAVING generation on shared memory could not be cancelled,
retried, or deleted, and it held its dataset name against generations and
uploads forever.

- Cancel accepts SAVING when this backend has no task for the generation.
  A save running in this backend still can't be cancelled.
- The dataset commit now completes the record with a conditional UPDATE on
  id, SAVING, and the expected execution id in the same transaction, and
  rolls back without inserting seeds when it matches nothing, so a cancel
  can never be overwritten by COMPLETED or leave seeds behind.
- The shared dataset name rule asks the database whether each requested
  name is taken and returns the names it matched, instead of re-checking
  stored spellings with casefold(). Width- or case-insensitive collations,
  such as SQL Server's default, now reject every variant they treat as
  equal, as the upload check did before.
- Batch members are checked one after another with earlier members
  written, so a batch with names the database treats as equal is rejected
  at creation.
- Correct the backend README.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Richard Lundeen (richlundeen) added a commit to richlundeen/PyRIT that referenced this pull request Nov 18, 2026
Remove three parts that staged ASSERT generation doesn't use:

- StorageIO.list_files_async (base, disk, and Azure Blob), the
  _list_directory_files helper, and the unquote in
  AzureBlobStorageIO._resolve_blob_name, with their tests. Blob-name
  handling returns to main's behavior.
- The reference-counted Azure Blob container client
  (_client_users, _acquire_client_async, _release_client_async) and its
  tests, restoring main's client handling. The client fix moves to its
  own FIX PR from main.
- TargetObjectiveProvider.REQUEST_METADATA_KEY, the stored
  target_objective_request metadata, request_from_metadata, and the
  instructions parameter of package_result. Generated objectives store
  the same metadata as on main. The stored request moves to microsoft#4134.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@romanlutz Roman Lutz (romanlutz) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs updating and resolving merge conflicts but otherwise looking good.

Staged generation (-4, microsoft#4133) merged to main with review changes, so a plain
merge would restore the old -4 content. This merge applies only this layer's
own changes onto main: it was computed against the tree this layer was built
on, old -4 1c0add0 plus main 8f3e5f4. `git diff main..HEAD` is the layer's
own diff plus the restored request recording below.

The -4 review moved the stored target-objective request to this PR. Restore
TargetObjectiveProvider.REQUEST_METADATA_KEY, the instructions argument of
package_result, request_from_metadata, and their tests, which the backend
adapter uses to rebuild a generation from its saved dataset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@richlundeen

Copy link
Copy Markdown
Contributor Author

review:5971128140

Merged current main (754eb44) and resolved the conflicts. Staged generation (-4) landed on main with review changes, so the merge applies only this PR's own changes on top of it. -4's review moved the stored target-objective request to this PR, so TargetObjectiveProvider again records the request and request_from_metadata reads it back, with their tests.

Verified on published commit 2b593415bb9ffe0818a71917e56cdbb20bf8e2dc.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ends

Review found that the generation jobs could lose or misreport work:

- A continue or cancel built from an older read wrote every column, so it
  could revert a review the API had already saved, or another backend's
  newer checkpoint. Each transition now writes only the fields it changes.
- The stored target-objective request had no owning value object and was
  re-read with the backend's create limits, so an SDK dataset with more
  than 1000 objectives returned 400 on Edit & regenerate, while create
  accepted blank harm categories that the generator then rejected.
  TargetObjectiveGenerationRequest now owns its namespaced metadata key and
  the generator's rules; the create limits stay on the create body, and an
  invalid stored value reads as unset. package_result takes an optional
  request, so main's callers keep working.
- Continue and retry required every recorded target, even for roles whose
  stages were done. They now resolve only required roles and roles with
  stages left, and check them with AssertGenerator.validate_checkpoint, the
  check advance_async already ran before any model call.
- Progress reported supplied ASSERT stages as still to run. It now follows
  the library's own stage plan from AssertGenerator.estimate.
- Batch members were ordered by 1 ms created_at offsets, which SQL Server
  DATETIME rounds together and which left updated_at before created_at.
  Members now share one created_at and store their batch position
  (group_index). Migration ccf97e771f04 adds the column after
  c4a401cd8201, which is already committed and so stays unchanged, and
  gives existing generations position 0.

Also removes the unused max_concurrent_generations option, corrects and
trims the README, and adds tests for the live-apply barrier, name holds
between generations, and a cancel whose run could not record it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Batch members share one created_at, and the list ordered them by batch
position descending, so the newest-first list showed a suite last member
first. The GUI groups that list by group_id and sorts by created_at, which
now ties, so it showed every suite reversed. Order members of one batch by
position ascending, in the query and in the chunked read.

Make try_update_prompt_generation_async require the fields it writes. Its
default wrote every column from the caller's snapshot, the stale overwrite
that field-scoped writes removed; only tests used it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.

A cancel releases the generation's dataset name, so another dataset or
generation can take it before a retry. The retry then returns 409 and the
checkpoint can't be saved under that name. Keep that behavior, but say so:
the retry and save conflicts now state that the generation can't resume or
save under the name and that a new generation under another name starts
over and repeats the model calls. The save conflict no longer points at
Edit & regenerate as if the work carried over. The README states the rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…equest rules

The dataset save is now one transaction that completes the record and inserts the
seeds, guarded by the run's execution ID. A separate "saving" status and the
service-wide write lock no longer protect anything, so a run stays running until
that transaction completes it. An in-process guard still keeps a local cancel or
shutdown from interrupting a save, and a cancel from another backend still makes
the save commit nothing. The memory writes were already compare-and-set in the
database, so the asyncio lock guarded no state.

Every caller reads one generation by ID and checks one dataset name, so the new
memory API takes one item: get_prompt_generation_async and
is_dataset_name_available_async, with is_dataset_name_taken_async as the shared
rule. The chunked reads, union query, Python re-sort, and multi-name error go.

TargetObjectiveProvider.package_result takes the request as its one source, so a
dataset's harm category labels can't disagree with the request it stores, and
TargetObjectiveGenerator checks requests through TargetObjectiveGenerationRequest
instead of repeating its rules. The signature came from -4 and is in no release.

The target-objective result summary is dropped: its objective_count always equals
the request's count and the progress.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.

@richlundeen
Richard Lundeen (richlundeen) added this pull request to the merge queue Nov 19, 2026
Merged via the queue into microsoft:main with commit 8a09795 Nov 19, 2026
69 of 71 checks passed
@richlundeen
Richard Lundeen (richlundeen) deleted the richlundeen-assert-generation-backend branch November 19, 2026 23:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants